Motion compensation in a three dimensional scan

ABSTRACT

The present disclosure provides computing device implemented methods, computing device readable media, and systems for motion compensation in a three dimensional scan. Motion compensation can include receiving three-dimensional (3D) scans of a dentition, estimating a motion trajectory from one scan to another, and calculating a corrected scan by compensating for the motion trajectory. Estimating the motion trajectory can include one or more of: registering a scan to another scan and determining whether an amount of movement between the scans is within a registration threshold; determining an optical flow based on local motion between consecutive two-dimensional (2D) images taken during the scan, estimating and improving a motion trajectory of a point in the scan using the optical flow; and estimating an amount of motion of a 3D scanner during the scan as a rigid body transformation based on input from a position tracking device.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/389,323, filed Apr. 19, 2019, now U.S. Patent Application PublicationNo. 2019/0244332, which is a continuation of U.S. patent applicationSer. No. 15/730,492, filed Oct. 11, 2017, now U.S. Pat. No. 10,275,862,which is a continuation of U.S. patent application Ser. No. 15/421,110,filed Jan. 31, 2017, now U.S. Pat. No. 9,830,688, which is acontinuation of U.S. patent application Ser. No. 14/970,030, filed Dec.15, 2015, now U.S. Pat. No. 9,589,329, which is a continuation of U.S.patent application Ser. No. 14/578,233, filed Dec. 19, 2014, now U.S.Pat. No. 9,214,014, which is a continuation of U.S. patent applicationSer. No. 13/666,812, filed Nov. 1, 2012, now U.S. Pat. No. 8,948,482,each of which is incorporated by reference in its entirety herein.

BACKGROUND

The present disclosure is related generally to the field of dentaltreatment. More particularly, the present disclosure is related tomethods, devices, and systems for motion compensation in a threedimensional scan.

Dental treatments may involve, for instance, restorative and/ororthodontic procedures. Restorative procedures may be designed toimplant a dental prosthesis (e.g., a crown, bridge inlay, onlay, veneer,etc.) intraorally in a patient. Orthodontic procedures may includerepositioning misaligned teeth and changing bite configurations forimproved cosmetic appearance and/or dental function. Orthodonticrepositioning can be accomplished, for example, by applying controlledforces to one or more teeth over a period of time.

As an example, orthodontic repositioning may be provided through adental process that uses positioning appliances for realigning teeth.Such appliances may utilize a thin shell of material having resilientproperties, referred to as an “aligner,” that generally conforms to apatient's teeth but is slightly out of alignment with a current toothconfiguration.

Placement of such an appliance over the teeth may provide controlledforces in specific locations to gradually move the teeth into a newconfiguration. Repetition of this process with successive appliances inprogressive configurations can move the teeth through a series ofintermediate arrangements to a final desired arrangement.

Such systems typically utilize materials that are light weight and/ortransparent to provide as a set of appliances that can be used seriallysuch that as the teeth move, a new appliance can be implemented tofurther move the teeth.

A scanning system may be used to obtain digital data representing apatient's teeth in their current position (i.e., at the time of thescan), which will be considered herein as an initial digital data set(IDDS) representing an initial tooth arrangement. The IDDS may beobtained in a variety of ways. This can be used for dental records ortreatment purposes.

For example, the patient's teeth may be imaged directly or indirectly(e.g., through the use of a model or impression) to obtain digital datausing direct and/or indirect structured light, X-rays, three-dimensionalX-rays, lasers, destructive scanning, computing device-aided tomographicimages or data sets, magnetic resonance images, intra-oral scanningtechnology, photographic reconstruction, and/or other imagingtechniques. The IDDS can include an entire mouth tooth arrangement,some, but not all teeth in the mouth, and/or it can include a singletooth.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe claims that follow. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIGS. 1A-1B illustrate alternate perspective views of an intraoralscanner according to a number of embodiments of the present disclosure.

FIG. 2 is a diagram illustrating motion of a scanner relative to anobject being scanned according to a number of embodiments of the presentdisclosure.

FIG. 3 is a diagram illustrating a scanner according to a number ofembodiments of the present disclosure.

FIG. 4 is a flow chart illustrating a method for motion compensationusing 3D and 2D data according to a number of embodiments of the presentdisclosure.

FIG. 5 is a flow chart illustrating a method for motion compensationusing 3D data according to a number of embodiments of the presentdisclosure.

FIG. 6 illustrates a system for motion compensation in a 3D scanaccording to a number of embodiments of the present disclosure.

DETAILED DESCRIPTION

A positive model and/or negative impression of the patient's teeth or atooth may be scanned, for example, using an X-ray, laser scanner,destructive scanner, structured light, and/or other range acquisitionsystem to produce the IDDS. The data set produced by the rangeacquisition system may be converted to other formats to be compatiblewith the software which is used for manipulating images within the dataset, as described herein.

Whether scanning a patient's dentition directly, or scanning a model orimpression of the patent's dentition, it may be desirable to supportscanning while the scanner is moving with respect to the teeth. Suchsupport can be beneficial for both video scanning (e.g., a continuoussequence of images) and point-and-shoot scanning (e.g., a number ofdiscrete images).

With point-and-shoot scanning, a user can trigger the scan. After thescan is triggered, a fast scan can be beneficial to reduce distortion inthe scan that may be caused by motion of the scanner that mightotherwise occur during a slow scan (e.g., during a time betweeninitiation and completion of the scan). It may be reasonable to assumethat movements associated with point-and-shoot scanning are relativelysmall (e.g., as compared to movements during video scanning) because theuser is trying to keep the scanner stable (e.g., as opposed to movingduring video scanning).

A slow scan can result in large movements of the scanner with respect tothe object during the scan, which can result in motion in the results ofthe scan (e.g., motion blur). As used herein, a slow scan does notindicate a long exposure during a scan. Rather, a slow scan can indicatethat a relatively long period of time is used to take multiple singleimages in the course of a single 3D scan. Thus, it can take a relativelylong time (e.g., 250-400 milliseconds (msec)) to capture a 3D scan. Eachimage on its own may or may not suffer motion blur. However, the effectof motion over the course of scans comprising the 3D scan can be thatthe relative position between raw images is unknown, thus causinggeometric distortion (e.g., due to interpretation of the raw images,which may assume a certain relative position between the images).However, a faster scan (e.g., 50 msec) can be more difficult toaccomplish (e.g., may call for more expensive and/or advanced hardwareand/or may call for more and/or faster computation capabilities versus aslower scan). Unless otherwise noted, the term scan, as used herein, canrefer to either or both of video and point-and-shoot scanning.

With continuous scanning, a scanner can automatically take a particularnumber of scans over a particular amount of time (e.g., x-number ofscans per second). It may be reasonable to assume a scanner velocity ofapproximately five millimeters (mm) per second (sec). However, even witha fast scan of 50 msec, this can still include a scanner drift of 250microns in some implementations. Thus, motion compensation can bebeneficial even for video scanning with fast scans.

As used herein, motion detection refers to a determination whether ascanner moved significantly with respect to the object. In someembodiments, scans with detected motion can be rejected. As used herein,motion compensation refers to correction for distortions in a scancaused by motion of the scanner. An example of motion compensation caninclude an estimation of a spatial trajectory of the scanner (e.g., howthe scanner moves in space) as a function of time in the duration ofeach single scan (e.g., as a function of time for each scan). Suchinformation can be used to compensate for distortions in a scan. In someembodiments, the result of a 3D scan can be a point cloud thatrepresents the scanned object (e.g., patient's dentition). Motioncompensation can be applied to the point cloud to correct for anydistortions resulting from the scan.

Some difficulties that may arise with motion compensation includecomplex motion (e.g., the scanner may move with six degrees of freedomand/or with an unsteady velocity). This may be particularly true forhand-held scanners that may contact the patient's teeth where frictionand/or collisions can cause a sudden acceleration. Other difficultiesinclude the patient's jaws moving during a scan. Some embodiments of thepresent disclosure include the use of position tracking devices in thescanner (e.g., an accelerometer and/or a gyroscope, among other inertialdevices, wireless devices, or other devices) to help measure movementsof the scanner, but such devices cannot account for movement of thepatient's jaws. Another difficulty can include motion of the soft tissue(e.g., tongue, lips, cheek) of the patient, which may confuse someoptical methods for computing motion of the scanner relative to theteeth.

The present disclosure provides computing device implemented methods,computing device readable media, and systems for motion compensation ina three dimensional scan. Motion compensation can include receivingthree-dimensional (3D) scans of a dentition, estimating a motiontrajectory from one scan to another, and calculating a corrected scan bycompensating for the motion trajectory. Estimating the motion trajectorycan include one or more of: registering a scan to another scan anddetermining whether an amount of movement between the scans is within aregistration threshold; determining an optical flow based on localmotion between consecutive two-dimensional (2D) images taken during thescan, estimating and improving a motion trajectory of a point in thescan using the optical flow; and estimating an amount of motion of a 3Dscanner during the scan as a rigid body transformation based on inputfrom a position tracking device

In some embodiments, motion compensation can include receiving firstdata from a 3D scan of a dentition taken with a 3D scanner in a firstreference frame and receiving second data from two-dimensional (2D)images of the dentition taken during the 3D scan of the dentition with a2D imager in a second reference frame that is fixed with respect to thefirst reference frame, where each of the 2D images is associated with arespective time in the 3D scan. Motion compensation can includedetermining an optical flow based on local motion between consecutiveimages of the 2D images, estimating a motion trajectory of a point inthe 3D scan, and improving the estimated motion trajectory using theoptical flow to estimate a correctness of the estimated motiontrajectory. Motion can be compensated for during the 3D scan using theimproved estimated motion trajectory.

In a number of embodiments, a system for motion compensation can includea scanning module including an intraoral 3D scanner and at least oneposition tracking device coupled to the 3D scanner. The system can alsoinclude a processing module coupled to the scanning module to estimatean amount of motion of the 3D scanner during a 3D scan as a rigid bodytransformation based on input from the at least one position trackingdevice and compensate for the amount of motion by applying an inverse ofthe rigid body transformation to a plurality of points comprising the 3Dscan.

In the detailed description of the present disclosure, reference is madeto the accompanying drawings that form a part hereof, and in which isshown by way of illustration how one or more embodiments of thedisclosure may be practiced. These embodiments are described insufficient detail to enable those of ordinary skill in the art topractice the embodiments of this disclosure, and it is to be understoodthat other embodiments may be utilized and that process, electrical,and/or structural changes may be made without departing from the scopeof the present disclosure. As used herein, the designator “N”,particularly with respect to reference numerals in the drawings,indicates that a number of the particular feature so designated can beincluded. As used herein, “a number of” a particular thing can refer toone or more of such things (e.g., a number of teeth can refer to one ormore teeth).

The figures herein follow a numbering convention in which the firstdigit or digits correspond to the drawing figure number and theremaining digits identify an element or component in the drawing.Similar elements or components between different figures may beidentified by the use of similar digits. For example, 104 may referenceelement “04” in FIG. 1, and a similar element may be referenced as 204in FIG. 2. As will be appreciated, elements shown in the variousembodiments herein can be added, exchanged, and/or eliminated so as toprovide a number of additional embodiments of the present disclosure. Inaddition, as will be appreciated, the proportion and the relative scaleof the elements provided in the figures are intended to illustratecertain embodiments of the present invention, and should not be taken ina limiting sense.

FIGS. 1A-1B illustrate alternate perspective views of an intraoralscanner according to a number of embodiments of the present disclosure.The spatial relationship of the scanner 106 with respect to thedentition 104 in the intraoral cavity is shown. A plurality of imagesshowing the relationship at any other desired vantage point (viewpoint)may be provided on a display.

Images presented on a display to a user can be composites of virtualmodels of the dentition 104 ac created with aid of the scanner 106 andstored in memory (see, for example, FIG. 7). These virtual models can bemanipulated by the user and/or the computing device to provide thecorrect spatial relationship, in virtual space and can be displayed as2D or 3D images. Optionally, the position of the scanner 106 and thedirection of the scanning axis (e.g., z-axis) can be displayed withrespect to the dentition 104. The scanning axis (e.g., z-axis) can bedefined as orthogonal to the scanning face 108 of the scanner 106, butmay be defined according to other suitable geometric or other parametersof the scanner 106. The images of the dentition 104 can be displayed ashaving 3D attributes and realistic dental morphologies and/or eachdental surface can be represented, for example, by a geometrical form(e.g., simple wedges representing incisors, cones representing canines,and cylinders representing molars).

The virtual model can be custom-modified to show a virtual preparationat each corresponding dental site where a real preparation is to befound. Virtual teeth may be removed from the model where none are to befound in the real intraoral cavity (e.g., where real teeth have beenremoved subsequent to obtaining the IDDS). These features can furtherfacilitate identification of the positions and orientations of thescanner 106 with respect to the dentition 104.

FIG. 2 is a diagram illustrating motion of a scanner 206 relative to anobject 210 being scanned according to a number of embodiments of thepresent disclosure. The result of scanning (e.g., after some processingof the scanned data) is a collection of points (x, y, z, t), where t isa time of focus and x, y, and z are 3D coordinates with respect to thecoordinate system of the scanner 206 at time t. Data that mightotherwise be captured by a scanner 206 may be lost due to motion driftof the scanner 206 during scanning. For example, walls above a certainangle may be invisible to the scanner 206 and/or areas near verticalwalls may be invisible.

Assuming a constant lateral speed (v) of the scanner 206 and assuming aconstant speed of scanning,

${{\tan\alpha} = \frac{\Delta z}{v\Delta t}},$where α is the drift angle (e.g., in degrees), Δz is the distancebetween the scanner 206 and the object 210 (e.g., in mm), v is thevelocity of the scanner 206 (e.g., in mm/sec), and Δt is the duration ofthe scan (e.g., in msec). The following table includes some exampledrift angles for Δz=15 mm

TABLE 1 α (degrees) Δt (msec) v = 5.0 mm/sec V = 10.0 mm/sec 50 89 88100 88 86 150 87 84 200 86 82 250 85 80

Missing data (e.g., due to the drift angle described above) can bedefined as

${{\Delta x} = {\frac{\Delta h}{\tan\alpha} = {v\frac{\Delta h\Delta t}{\Delta z}}}},$where Δx is the width of the missing portion (e.g., in microns) and Δhis the height of the missing portion (e.g., in mm) The following tableincludes some example of amounts of missing data for Δz=15 mm and Δh=10mm

TABLE 2 Δx (microns) Δt (msec) v = 5.0 mm/sec V = 10.0 mm/sec 50 166 333100 333 666 150 500 1000 200 666 1333 250 833 1666

Because of imperfections in scanning technique and assuming general raygeometry, the scan may not have telecentric properties (e.g., havingrays entering the scanner 206 parallel to the z-axis), the focal surfaceis not necessarily a plane (e.g., a flat surface may appear curvedand/or a non-flat object 210 represented in the image as a focal surfacemay not be represented in a plane as it might otherwise), and the scanmay include arbitrary scanning patterns (e.g., the z-axis may not beconsistent with respect to the object during the scan, that is, thescanner 206 may not be consistently pointed directly at the object 210during the scan). This can result in a distorted image that appears tobow and/or curve where it shouldn't, among other distortions. Forexample, a rectangle might appear as a parallelogram, perhaps withcurved sides.

According to various embodiments of the present disclosure, motionestimation can include any one of, or any combination of approximationbased on registration results, image processing with optical flow, theuse of position tracking devices with the scanner 206, differentialsignaling between the scanner 206 and a plurality of wirelesstransmitters, and/or the use of any other motion estimation/positiontracking technique. Use of registration results can measure motionrelative to the teeth, is relatively easy to compute, and is generallynot confused by soft tissue, but may have inaccuracies due to assumptionof a simplified motion trajectory and may have performance limited basedon a scanning rate. Use of image processing with optical flow canmeasure motion relative to the teeth and can have relatively highaccuracy, but may be computationally intensive and may be confused bysoft tissues. Use of position tracking devices is relatively easy tocompute but may be confused by gravitation and may be indifferent topatient movements.

FIG. 3 is a diagram illustrating a scanner 306 according to a number ofembodiments of the present disclosure. For example, the scanner 306 canbe a hand-held intraoral scanner. The scanner 306 can include a 3Dscanner 312. By way of example, the 3D scanner 312 can determine a 3Dstructure by confocal focusing of an array of light beams, as will beappreciated by those of ordinary skill in the art. The scanner 306 caninclude only a 3D scanner 312, or a 3D scanner 312 in combination withone or more of a 2D imager 314 and a number of position tracking devices320-1, . . . , 320-N. The position tracking devices 320-1, . . . , 320-Ncan include, for example, an accelerometer (e.g., to measureacceleration of the scanner 306) and a gyroscope (e.g., to measureangular velocity of the scanner 306), among others. Although notspecifically illustrated, the scanner 306 can include and or be coupledto a computing device (e.g., including a processor and memory as isillustrated in FIG. 7).

The 2D imager 314 can be analogous to a digital camera. The 2D imager314 can be configured to continuously capture discrete images of theobject while the 3D scanner 312 is scanning the object. The 2D imager312 can be fixed with respect to the 3D scanner 312 such that anymovement of the scanner 306 equally affects the 2D imager 312 and the 3Dscanner 312 (e.g., reference frame of the 2D imager 312 can be fixedwith respect to a reference frame of the 3D scanner). Furthermore, the2D imager 312 can be non-concentric with respect to the 3D scanner 312(e.g., to improve sensitivity to motion along the z-axis).

As is illustrated in FIG. 3, the position of a focal point of a singlebeam of light 318 from the 3D scanner 312 is not vertical because thescanner 306 is moving during the scan. The scanner 306 can move along atrajectory 322 such that the scanner 306, during a time of capture ofthe 3D scan (e.g., Δt) the focal plane can move a particular distance(e.g., Δz). The result of 3D scanning (e.g., after some processing ofthe scanned data) can be a collection of points 316. In some instances,reconstruction of the trajectory 322 can be limited to a constant-speedtrajectory.

In some embodiments, motion of the 3D scanner 312 can be tracked basedonly on input from the position tracking devices 320-1, . . . , 320-Nand input from the 2D imager 314 (e.g., without using input from the 3Dscanner 312). Such motion tracking may be based on an assumption thatthe patient's jaw doesn't change its orientation. The position trackingdevices 320-1, . . . , 320-N can provide information indicating a changein orientation of the 3D scanner 312 (e.g., in three degrees offreedom). The analysis of 2D images can provide information indicating achange in orientation in the other three degrees of freedom.

In various embodiments, a velocity of the 3D scanner 312 can beestimated based on input from the position tracking devices 320-1, . . ., 320-N. The estimated velocity can be reset to zero in response toinput from the 2D imager 314 indicating that the 3D scanner 312 is notmoving. For example, the estimated velocity can be reset to zero in sixdegrees of freedom in response to the input from the 2D imager 314comprising a plurality of 2D images indicating that the 3D scanner 312is not moving. Such embodiments can be beneficial in avoiding largeerror drifts. If the user keeps the scanner 306 stable from time totime, the 2D images can be used to determine when the scanner 306 isheld in the stable position.

FIG. 4 is a flow chart illustrating a method for motion compensationusing 3D and 2D data according to a number of embodiments of the presentdisclosure. A scan 424 can be initiated. A first input to an imageprocessing algorithm can be data from a 3D scan 428 (e.g., a height mapthat is a result of some initial processing of data gathered by the 3Dscanner) of a dentition taken with the 3D scanner in a first referenceframe. In some embodiments, the 3D data can result from a confocal scanin the z-direction.

A second input to the image processing algorithm can be a plurality of2D images 426 of the dentition (e.g., a video sequence) taken during the3D scan of the dentition with a 2D imager (e.g., a secondary camera) ina second reference frame that is fixed with respect to the firstreference frame. In some embodiments, the 2D imager can have a highdepth of field, which can help provide identical or nearly identicalquality of motion estimation throughout the entire depth of scanning. Insome embodiments, the second reference frame can be non-telecentric forimproved sensitivity to motion along the z-axis. Each of the 2D imagescan be associated with a respective time in the 3D scan (e.g., each ofthe 2D images can be captured and time stamped in the duration of the 3Dscan).

In some embodiments, an heuristic can be used to determine which pointsin the 3D scan are visible in a particular one of the plurality of 2Dimages (e.g., prior to determining an optical flow). If a 3D point isoccluded at a certain time, then checking the optical flow along itspath could lead to inaccurate results. Accordingly, 3D points that areoccluded and/or that go out of the field-of-view in each 2D image can befiltered out from the analysis.

The result of 3D scanning 430 (e.g., after some processing of thescanned data) can be a collection of points (x, y, z, t), where t is atime of focus and x, y, and z are 3D coordinates with respect to thecoordinate system of the 3D scanner at time t. A 2D image can include anumber of pixels with coordinates (i, j, t), where i and j arecoordinates of the pixel on a sensor of the 2D imager and t correspondsto a time in the 3D scan that the 2D image was taken. The coordinates x,y, and z can be calculated from (i, j, t) where i and j are indices of apixel on a sensor of the scanner. In some embodiments, the mapping from(i, j, t) to (x, y, z) can be a constant map from 3D to 3D that takesinto consideration constant geometrical distortions, but assumed thatthe scanner is not moving. A point (x, y, z) in the coordinate system ofthe 3D scanner can be mapped to pixel coordinates (i, j) of the 2Dimager. The time (t) is not necessarily proportional to z as in confocalsystems. For example, a scan can be made by moving the x-axis slowly andthen t would be proportional to x. However, in some embodiments that useconfocal imaging, t can be proportional to z.

An optical flow can be determined based on local motion betweenconsecutive 2D images. The optical flow is an observation of the localmotion at each area of the image, between two consecutive images in thesequence. The optical flow can be determined by calculating the localmotion between the two consecutive images based on a change in positionof a point in the two consecutive images and based on a change in timebetween the two consecutive images. In some embodiments the local motioncan be calculated as a 2D motion vector based on a change in position ofthe point.

For example, given two images at consecutive times, I(t) and I(t+dt), dxand dy can be found such that I(i, j, t)=I(i+dx, j+dy, t+dt). Where vxand vy are components of a 2D velocity and where dx=dt*vx, dy=dt*vy,then I(i, j, t)=I(i+dt*vx, j+dt*vy, t+dt). The components vx and vy candepend upon i, j, and t because the 2D motion vector can vary across theimage and vary in time. Using a Taylor expansion: I(i+dt*vx, j+dt*vy,t+dt)=I(i, j, t)+dt*vx*dI/dx+dt*vy*dI/dy+dt*dI/dt=I(x, y, t). Therefore,the optical flow equation can be O(vx, vy, i, j,t)=vx*dI/dx+vy*dI/dy+dI/dt=0.

A motion trajectory 432 of a point in the 3D scan can be estimated. Forexample, the algorithm can find a motion trajectory that best explainsthe optical flow of the video sequence given the 3D data. Estimating themotion trajectory can include using a rigid body transformation of thepoint dependent upon time in the 3D scan. For example, the rigid bodytransformation can be defined such that applying the rigid bodytransformation to the point maps the point from a coordinate system at areference time in the 3D scan to a coordinate system at a recorded time(e.g., time t) of the point in the 3D scan. Non-limiting examples of thereference time can include time 0 (e.g., an initial time of the scan),an end time of the scan, and a middle time of the scan, among othertimes during the scan. The coordinate system of the scanner is movingalong a trajectory T(t), where T(t) is a rigid body transformation thatmaps (x, y, z) at a recorded time in the scan (e.g., time t) to (x, y,z) at a reference time in the scan (e.g., time 0, an initial time). Thatis, a point (x, y, z) in the coordinate system of the scanner at time tcan be represented as the point T(t)*(x, y, z) in the coordinate systemof the scanner at time t=0. A point (x1, y1, z1, t1) can be mapped tothe coordinate system at time t0 b applying inverse (T(t0))*T(t1). Agoal of motion estimation can be to approximate T(t). Motioncompensation 434 can be performed by application of T(t) to points (x,y, z, and t).

Estimating the motion trajectory can include, as an initial guess,estimating a global velocity vector (e.g., (vx, vy) for each consecutiveimage of the 2D imager I(t), I(t+dt)) based only on the optical flow(e.g., without considering the scan data) and mapping the globalvelocity vector to a lateral motion trajectory (e.g., T(t)), where thelateral motion trajectory is the estimated motion trajectory. In someembodiments, an position tracking device (e.g., an accelerometer) can beused as an additional input source to provide information to enhance theinitial guess. In some embodiments, passive and/or uniform lighting maybe assumed for the 2D imager and/or that the 2D imager can be calibratedwith respect to the coordinates of the height map.

The estimated motion trajectory can be improved using the optical flowto estimate a correctness of the estimated motion trajectory. For eachpoint (x1, y1, z1, t1) in the scan data, the motion trajectory can beused to find its position in the coordinate system of the scanner attimes t, (x(t), y(t), z(t), t). For example, an objective function canbe defined that evaluates the correctness of a given motion trajectoryaccording to the optical flow in the video sequence. The 3D data can beused for mapping points to 2D paths in the video sequence. In someembodiments, as an initial guess, an assumption can be made that thereis no motion or a simple motion trajectory can be used.

In some embodiments, estimating the correctness of the estimated motiontrajectory can include mapping the point from a coordinate system of the3D scan to a coordinate system of the 2D scan to result in a 2Dtrajectory for the point. The resulting 2D trajectory (e.g., (i(t),j(t), t)) can de differentiated to determine a number of velocityelements (e.g., vx(t), vy(t)) of the point. The 2D trajectory and thenumber of velocity elements can be applied to the optical flow (e.g.,O(vx(t), vy(t), i(t), j(t), t)=0 for all t). The estimated motiontrajectory can be improved by minimizing an objective function thatpenalizes inaccurate trajectories (e.g., at each point in the 3D scan)based on the optical flow with the applied 2D trajectory and the numberof velocity elements. A sum of the squares of the optical flow with theapplied 2D trajectory and the number of velocity elements can becalculated between a reference time in the 3D scan and a recorded timeof the point in the 3D scan (e.g., O(vx(t), vy(t), i(t), j(t), t) overall points (x1, y1, z1, t1) and all times t). The sum of the squares ofthe optical flow with the applied 2D trajectory and the number ofvelocity elements can be minimized to improve the estimated motiontrajectory (e.g., Minimize Sum (O(vx(t), vy(t), i(t), j(t),t){circumflex over ( )}2) over all points (x1, y1, z1, t1) and all timest).

In some embodiments, the 2D trajectory can be regularized to penalizeinaccurate trajectories such as unexpected motion patterns, suddenjumps, and/or discontinuities and to stabilize the numerical procedure.The 2D trajectory can be regularized to limit the number of velocityelements within a particular range of velocities. Such a range may bebased on a priori knowledge about the motion trajectory that limits itto a reasonable range, penalizes sudden “jumps” between consecutivetransformation T(t) and T(t+dt), and/or increases the stability of thenumerical procedure. For example, the regularization can be: MinimizeRegularization_Term(T(j))+Sum(O(vx(t), vy(t), i(t), j(t), t){circumflexover ( )}2) over all points (x1, y1, z1, t1) in the scan data over allt.

The improved estimated motion trajectory can be used to compensate formotion during the 3D scan. For example, compensating for motion duringthe 3D scan can include applying an inverse of the rigid bodytransformation to the point in the 3D scan to map the point from therecorded time of the point in the 3D scan to the reference time (e.g.,time 0, an initial time) in the 3D scan (e.g., time t). That is, givenT(t), the 3D scan data can be corrected by applying T(t) to each point(x, y, z, t) to bring each point (x, y, z, t) from the coordinate systemof the scanner at time t to the coordinate system of the scanner at time0.

In various embodiments, motion compensation functionality can be testedusing only the 2D imager (e.g., where the 3D scanner is non-functional).A special test object can be prepared for scanning with visible fiducialmarkings and digitized with the markings. The 2D imager can becalibrated to find the relationship between (x, y, z) coordinates andimage coordinates (i, j) in some coordinate system. A sequence of imagescan be captured while the camera is moved manually over the object. Themotion trajectory can be computed from the sequence by, for each imagein the sequence, detecting the markings in the image and computing theposition of the camera with respect to the object based on the cameracalibration. From the motion trajectory, a height map can be calculated,assuming a particular pattern of scanning in the z-direction. From this,the height map and sequence of images can be fed into the previouslydescribed algorithm and the results can be tested against the calculatedmotion trajectory.

FIG. 5 is a flow chart illustrating a method for motion compensationusing 3D data according to a number of embodiments of the presentdisclosure. Multiple consecutive 3D scans can be used for motioncompensation. A sequence of 3D scans 536 can be taken (e.g., at a rateof at least ten 3D scans per second). For example, the 3D scans can be3D intraoral scans of a patient's dentition. The 3D scans can be saved538 and distorted scans 540 (e.g., S(n)) can be registered 542 to aprevious scan (e.g., an immediately previous scan S(n−1)) and/or to asubsequent scan (e.g., an immediately subsequent scan S(n+1)). In anumber of embodiments, a distorted scan can be registered to a pluralityof previous 3D scans and/or to a plurality of subsequent 3D scans. Insome embodiments, there may be multiple distorted scans (e.g., everyscan). Thus, the term “distorted scan” with respect to the descriptionof the use of registration results merely identifies a particular scanthat can be used along with a previous scan and a subsequent scan formotion compensation. In various embodiments, a subsequent or previousscan may not be distorted, which can improve correction of the distortedscan by registering it to a scan that is not distorted.

In some embodiments, registration can include aligning the previous scanwith respect to the distorted scan and/or aligning the subsequent scanwith respect to the distorted scan and determining an overlappingportion of the previous scan with respect to the distorted scan and/ordetermining an overlapping portion of the subsequent scan with respectto the distorted scan based on the respective alignment. Accuracy ofoverlap determined based on comparison between surfaces can be dependenton the shape of the object being scanned. For example, an increasednumber of z-heights that are common to both a first and a second scancan improve the motion estimation that is calculated based at least inpart on comparison of scan surfaces.

Aligning two different scans can include the use of a rigid bodytransformation (e.g., R(n−1:n)) on the previous scan (e.g., S(n−1)) toalign it with the distorted scan (e.g., S(n)) and/or a rigid bodytransformation (e.g., R(n+1:n)) on the subsequent scan (e.g., S(n+1)) toalign it with the distorted scan (e.g., S(n)). The rigid bodytransformation is a six degrees of freedom transformation. An amount ofmovement can be determined between the previous scan and the distortedscan and/or between the distorted scan and the subsequent scan based onthe respective overlapping portion and/or based on the respectivenon-overlapping portion.

A determination 544 can be made as to whether an amount of movementbetween the previous scan, the distorted scan, and the subsequent scan(e.g., based on the respective overlapping portion and/or based on therespective non-overlapping portion) is within a registration threshold.If the amount of movement is not within the threshold (e.g., if thescanner moved too much during the scan), then the distorted scan can berejected 546. If the amount of movement is within the registrationthreshold, then a first motion trajectory 548 can be estimated from theprevious scan to the subsequent scan based on the registration (e.g.,T(n−1:n+1) from S(n−1) to S(n+1)). The registration can be used toprovide knowledge of the relative positions during the previous,distorted, and subsequent scans.

The registration results can signify snapshots of the trajectory T(t . .. i) at a finite number of times (t . . . i) and the motion trajectorycan be computed from those snapshots by an interpolation method. Thenumber of points (t . . . i) and the interpolation may vary. In someembodiments the first motion trajectory can be a snapshot of an overallmotion trajectory across a plurality of 3D scans. The overall motiontrajectory can be calculated based on a plurality of snapshots eachassociated with a respective registered one of the plurality of 3Dscans.

A second motion trajectory 550 can be estimated during a scanning timeof the distorted scan. The second motion trajectory can be a segment ofthe first motion trajectory (e.g., T(n−1:n+1)) during S(n). The secondmotion trajectory can be estimated in response to the amount of movementbeing within the registration threshold.

In some embodiments, registration can include associating a first pointon a time axis in a middle of a scanning time of a previous scan with arigid body transformation of the previous scan that aligns the previousscan with the distorted scan. A second point on the time axis in amiddle of a scanning time of the subsequent scan can be associated witha rigid body transformation of the subsequent scan that aligns thesubsequent scan with the distorted scan. Estimating the first motiontrajectory can include interpolating between the first rigid bodytransformation and the second rigid body transformation. Estimating thesecond motion trajectory can include trimming the first motiontrajectory to only include motion trajectory during the scanning time ofthe distorted scan.

A corrected scan 552 (e.g., S*(n)) can be calculated for the distortedscan by compensating for the motion trajectory during the scanning timeof the distorted scan. The scan can be corrected in response to theamount of movement being within the registration threshold. Thus, insome embodiments, using registration, motion compensation can beperformed without the use of extra data for motion estimation (e.g.,position tracking devices and/or a 2D imager). However, such extra datacan be included to further improve motion compensation as describedherein. In various embodiments, the registration of the distorted scan,estimation of the first motion trajectory, estimation of the secondmotion trajectory, and calculation of the corrected scan can beiteratively repeated to improve an accuracy of the corrected scan.Iterative repetition can improve an accuracy of the scans. Aftercorrecting for distortion, the registration itself can become moreaccurate, which can improve the next iterative correction. The iterativerepetition can continue until the results converge, for example.

FIG. 6 illustrates a system for motion compensation in a 3D scanaccording to a number of embodiments of the present disclosure. Thesystem illustrated in FIG. 6 can include a computing device 680 having anumber of components coupled thereto. The computing device 680 caninclude a processing module 681 (e.g., one or more processors) andmemory 682. The memory 682 can include various types of informationincluding data 683 and executable instructions 684 as discussed herein.The memory 682 can be a tangible non-transitory computer readable mediumhaving instructions 684 that can be executed by the processing module681 to cause the computing device 680 to perform various functions asdescribed herein.

The memory 682 and/or the processing module 681 may be located on thecomputing device 680 or off the device in some embodiments. As such, asillustrated in the embodiment of FIG. 6, a system can include a networkinterface 685. Such an interface can allow for processing on anothernetworked computing device or such devices can be used to obtaininformation about the patient or executable instructions for use withvarious embodiments provided herein.

As illustrated in the embodiment of FIG. 6, a system can include one ormore input and/or output interfaces 686. Such interfaces can be used toconnect the computing device 680 with one or more input or outputdevices.

For example, in the embodiment illustrated in FIG. 6, the systemincludes connectivity to a scanning module 606 (e.g., an intraoralscanner including one or more of a 3D scanner, a 2D imager, and/or anumber of position tracking devices as described herein), a camera dock688, an input device 689 (e.g., a keyboard, mouse, etc.), a displaydevice 690 (e.g., a monitor), and a printer 691. The input/outputinterface 686 can receive data, storable in the data storage device(e.g., memory 682), representing 3D and/or 2D data, among other data,corresponding to the patient's dentition. Although the scanning module606 is illustrated as a component separate from the computing device680, in some embodiments, the scanning module 606 (e.g., the intraoralscanner) can include the components and/or functionality describedherein with respect to the computing device 680 (e.g., the scanningmodule 606 can perform motion detection and/or motion compensation).

In some embodiments, the scanning module 606 can be configured to scan aphysical mold of a patient's upper jaw and a physical mold of apatient's lower jaw. In one or more embodiments, the scanning module 606can be configured to scan the patient's upper and/or lower jaws directly(e.g., intraorally).

The camera dock 688 can receive an input from an imaging device (e.g., a2D imaging device) such as a digital camera or a printed photographscanner separate from any 2D imaging device on the scanning module 606.The input from the imaging device can be stored in the data storagedevice 682.

The processing module 681 can be configured to provide a visualindication of a virtual dental model on the display 690 (e.g., on a GUIrunning on the processing module 681 and visible on the display 690).The processing module 681 can further be configured (e.g., via computerexecutable instructions stored in a tangible non-transitory computerreadable medium) to perform the various methods, algorithms, and/orfunctionality described herein. The processing module 681, inassociation with the data storage device 682, can be associated withdata and/or application modules 692. The processing module 681, inassociation with the data storage device 682, can store and/or utilizedata and/or execute instructions to provide a number of applicationmodules for motion compensation in a 3D scan.

Such connectivity can allow for the input and/or output of virtualdental model information or instructions (e.g., input via keyboard)among other types of information. Although some embodiments may bedistributed among various computing devices within one or more networks,such systems as illustrated in FIG. 6 can be beneficial in allowing forthe capture, calculation, and/or analysis of information discussedherein.

A system for motion compensation can include a scanning module 606 and aprocessing module 681). The scanning module 606 can include an intraoral3D scanner and, in some embodiments, at least one position trackingdevice coupled to the 3D scanner. Position tracking devices can includeone or more of an accelerometer (e.g., to measure acceleration of the 3Dscanner), a gyroscope (e.g., to measure angular velocity of the 3Dscanner), among others. Other position tracking devices can include theuse of triangulation, trilateration, or multilateration to determineposition, for example using Wi-Fi, Bluetooth, or other wireless systems.

The processing module 681 can (e.g., via application module 692) receivea plurality of 3D scans of a dentition and estimate a motion trajectoryfrom a particular one of the plurality of scans to a subsequent scan.The estimation can include one or more of registering the particularscan to at least one of a previous scan and the subsequent scan,determining whether an amount of movement between the previous scan, theparticular scan, and the subsequent scan is within a registrationthreshold, and estimating the motion trajectory based on theregistration, using data from a plurality of two-dimensional (2D) imagesof the dentition taken during the particular scan of the dentition,determining an optical flow based on local motion between consecutiveimages of the plurality of 2D images, estimating a motion trajectory ofa point in the particular scan, improving the estimated motiontrajectory using the optical flow to estimate a correctness of theestimated motion trajectory, and estimating 654 an amount of motion ofthe 3D scanner during a 3D scan as a rigid body transformation based oninput from the at least one position tracking device. The processingmodule 681 (e.g., via application module 692) can calculate a correctedscan by compensating for the motion trajectory according to any of thetechniques described herein.

The processing module 681 can estimate the motion trajectory andcalculate the corrected scan in response to the amount of movement beingwithin the registration threshold, and to reject the particular scan inresponse to the amount of movement being outside of the registrationthreshold. The processing module 681 can register the particular scan toa plurality of previous scans and to a plurality of subsequent scans.

The processing module 681 can align the previous scan with respect tothe particular scan, determine an overlapping portion of the previousscan with respect to the particular scan based on the alignment, anddetermine an amount of movement between the previous scan and theparticular scan based on the overlapping portion and/or anon-overlapping portion to register the particular scan to the at leastone of the previous scan and the subsequent scan. The motion trajectorycan be a snapshot of an overall motion trajectory across the pluralityof scans, and where the instructions can be executed by the processor tocompute the overall motion trajectory based on a plurality of snapshotseach associated with a respective registered one of the plurality ofscans.

The at least one position tracking device can be coupled to the 3Dscanner and the processing module 681 can be configured to compensate656 for the estimated amount of motion by applying an inverse of therigid body transformation to a plurality of points comprising theparticular scan. A 2D imager can be coupled to the 3D scanner and theprocessing module 681 can estimate a velocity of the 3D scanner based oninput from the at least one position tracking device and to reset theestimated velocity to zero in response to input from the 2D imagerindicating that the 3D scanner is not moving. The processing module 681can reset the estimated velocity to zero in six degrees of freedom inresponse to the input from the 2D imager comprising the plurality of 2Dimages indicating that the 3D scanner is not moving.

The 3D scanner can be in a first reference frame and the scanning module606 can include a 2D imager in a second reference frame that is fixedwith respect to the first reference frame. The processing module 681 canreceive the data from the plurality of 2D images of the dentition taken,with the 2D imager, during a particular scan of the dentition, whereeach of the plurality of 2D images is associated with a respective timein the particular scan. The processing module 681 can compensate formotion during the scan using the improved estimated motion trajectory.

The processing module 681 can calculate the local motion between twoconsecutive images based on a change in position of a point in the twoconsecutive images and based on a change in time between the twoconsecutive images to determine the optical flow. The processing module681 can estimate a global velocity vector based only on the opticalflow. The processing module can map the global velocity vector to alateral motion trajectory comprising the estimated motion trajectory.

The processing module 681 can map the point from a coordinate system ofthe scan to a coordinate system of the image to result in a 2Dtrajectory for the point. The processing module 681 can differentiatethe resulting 2D trajectory to determine a number of velocity elementsof the point. The processing module 681 can apply the 2D trajectory andthe number of velocity elements to the optical flow. The processingmodule can sum squares of the optical flow with the applied 2Dtrajectory and the number of velocity elements between a reference timein the scan and a recorded time of the point in the scan to estimate thecorrectness of the estimated motion trajectory.

The processing module 681 can minimize an objective function thatpenalizes inaccurate trajectories at each point in the scan based on theoptical flow with the applied 2D trajectory and the number of velocityelements to improve the estimated motion trajectory. The processingmodule 681 can regularize the 2D trajectory to penalize inaccuratetrajectories.

Although specific embodiments have been illustrated and describedherein, those of ordinary skill in the art will appreciate that anyarrangement calculated to achieve the same techniques can be substitutedfor the specific embodiments shown. This disclosure is intended to coverany and all adaptations or variations of various embodiments of thedisclosure.

It is to be understood that the use of the terms “a”, “an”, “one ormore”, “a number of”, or “at least one” are all to be interpreted asmeaning one or more of an item is present. Additionally, it is to beunderstood that the above description has been made in an illustrativefashion, and not a restrictive one. Combination of the aboveembodiments, and other embodiments not specifically described hereinwill be apparent to those of skill in the art upon reviewing the abovedescription.

It will be understood that when an element is referred to as being “on,”“connected to” or “coupled with” another element, it can be directly on,connected, or coupled with the other element or intervening elements maybe present. In contrast, when an element is referred to as being“directly on,” “directly connected to” or “directly coupled with”another element, there are no intervening elements or layers present. Asused herein, the term “and/or” includes any and all combinations of oneor more of the associated listed items.

It will be understood that, although the terms first, second, etc. maybe used herein to describe various elements and that these elementsshould not be limited by these terms. These terms are only used todistinguish one element from another element. Thus, a first elementcould be termed a second element without departing from the teachings ofthe present disclosure.

The scope of the various embodiments of the disclosure includes anyother applications in which the above structures and methods are used.Therefore, the scope of various embodiments of the disclosure should bedetermined with reference to the appended claims, along with the fullrange of equivalents to which such claims are entitled.

In the foregoing Detailed Description, various features are groupedtogether in a single embodiment for the purpose of streamlining thedisclosure. This method of disclosure is not to be interpreted asreflecting an intention that the embodiments of the disclosure requiremore features than are expressly recited in each claim.

Rather, as the following claims reflect, inventive subject matter liesin less than all features of a single disclosed embodiment. Thus, thefollowing claims are hereby incorporated into the Detailed Description,with each claim standing on its own as a separate embodiment.

What is claimed is:
 1. A system, comprising: a non-transitory memoryhaving computer-readable instructions for compensating for motion in athree-dimensional (3D) scan using two-dimensional (2D) images collectedduring the 3D scan; and a processor coupled to the non-transitory memoryand configured to execute the computer-readable instructions to:determine an optical flow by calculating a local motion vector betweenconsecutive images of the 2D images based on a change in position of apoint in the consecutive images and a change in time between theconsecutive images; estimate a motion trajectory using a global velocityvector for each consecutive image of the 2D images based on the opticalflow; improve the estimated motion trajectory using the optical flow toestimate a correctness of the estimated motion trajectory; and calculatea corrected scan for a particular 3D scan by compensating for motionduring the particular 3D scan using the improved estimated motiontrajectory.
 2. The system of claim 1, further comprising instructions toenhance the motion trajectory using information from a position trackingdevice of a 3D scanner.
 3. The system of claim 1, wherein instruction toimprove the estimated motion trajectory include instructions to map aparticular point in a coordinate system of the particular 3D scan to acoordinate system of the 2D images resulting in a 2D trajectory for theparticular point.
 4. The system of claim 3, further comprisinginstructions to determine a number of velocity elements of theparticular point by differentiating the 2D trajectory, and apply the 2Dtrajectory and the number of velocity elements to the optical flow tominimize an objective function that penalizes inaccurate trajectories.5. The system of claim 4, further comprising instructions to calculate asum of the squares of the optical flow with the applied 2D trajectoryand the number of velocity elements between a reference time in the 3Dscan and a recorded time of the particular point in the 3D scan.
 6. Thesystem of claim 4, further comprising instructions to regularize the 2Dtrajectory to penalize inaccurate trajectories or to limit the number ofvelocity elements within a particular range of velocities.
 7. The systemof claim 1, further comprising instructions to register a plurality of2D images taken during the particular 3D scan.
 8. The system of claim 1,further comprising instructions to map the global velocity vector ofeach of the 2D images to the estimated motion trajectory.
 9. The systemof claim 1, further comprising instructions to register the particular3D scan by aligning one or more of a previous scan and a subsequent scanwith a distorted scan and to determine an overlap based on thealignment.
 10. The system of claim 1, further comprising instructions toestimate an amount of motion during the particular 3D scan by applying arigid body transformation of a particular point dependent upon time inthe particular 3D scan.
 11. The system of claim 10, wherein applying therigid body transformation to the particular point maps the particularpoint from a coordinate system at a reference time in the particular 3Dscan to a coordinate system at a recorded time of the particular pointin the particular 3D scan.
 12. A computer implemented method forcompensating for motion in three-dimensional (3D) scanning of adentition, comprising: determining an optical flow by calculating alocal motion vector between consecutive images of two-dimensional (2D)images collected during the 3D scan, the optical flow based on a changein position of a point in the consecutive images and a change in timebetween the consecutive images; estimating a motion trajectory using aglobal velocity vector for each consecutive image of the 2D images basedon the optical flow; improving the estimated motion trajectory using theoptical flow to estimate a correctness of the estimated motiontrajectory; and calculating a corrected scan for a particular 3D scan bycompensating for motion during the particular 3D scan using the improvedestimated motion trajectory.
 13. The method of claim 12, furthercomprising estimating an amount of motion during the particular 3D scanby applying a rigid body transformation of a particular point dependentupon time in the particular 3D scan.
 14. The method of claim 13, whereinapplying the rigid body transformation to the particular point maps theparticular point from a coordinate system at a reference time in theparticular 3D scan to a coordinate system at a recorded time of theparticular point in the particular 3D scan.
 15. The method of claim 12,further comprising estimating an amount of motion of a 3D scanner basedon input from at least one position tracking device.
 16. The method ofclaim 12, further comprising mapping a particular point in a coordinatesystem of the particular 3D scan to a coordinate system of the 2D imagesresulting in a 2D trajectory for the particular point.
 17. A dentalscanning system, comprising: a three-dimensional (3D) scanner configuredto collect 3D scans of a patient's dentition, the 3D scanner including atwo-dimensional (2D) imager configured to collect 2D images duringcollection of the 3D scans; a processor configured to executecomputer-readable instructions for: determining an optical flow bycalculating a local motion vector between consecutive images of the 2Dimages based on a change in position of a point in the consecutiveimages and a change in time between the consecutive images; estimating amotion trajectory using a global velocity vector for each consecutiveimage of the 2D images based on the optical flow; improving theestimated motion trajectory using the optical flow to estimate acorrectness of the estimated motion trajectory; and calculating acorrected scan for a particular 3D scan by compensating for motionduring the particular 3D scan using the improved estimated motiontrajectory.
 18. The dental scanning system of claim 17, wherein theinstructions include estimating an amount of motion during theparticular 3D scan by applying a rigid body transformation of aparticular point dependent upon time in the particular 3D scan.
 19. Thedental scanning system of claim 18, wherein applying the rigid bodytransformation to the particular point maps the particular point from acoordinate system at a reference time in the particular 3D scan to acoordinate system at a recorded time of the particular point in theparticular 3D scan.
 20. The dental scanning system of claim 17, theinstructions include registering the particular 3D scan by aligning oneor more of a previous scan and a subsequent scan with a distorted scanand determining an overlap based on the alignment.